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The Reform Support Network, sponsored by the U.S. Department of Education, supports the Race to 
the Top grantees as they implement reforms in education policy and practice, learn from each other, 
and build their capacity to sustain these reforms, while sharing these promising practices and lessons 
learned with other States attempting to implement similarly bold education reform initiatives. 


Introduction 


Over the past decade, national initiatives have shifted 
States’ attention to the challenge of turning around 
low-performing schools. Federal programs such as 
Race to the Top, School Improvement Grants (SIG) 

and Elementary and Secondary Education Act (ESEA) 
flexibility waivers have spurred States to increase the 
level of resources for and attention to persistently low- 
performing schools. While there have been numerous 
success stories of school turnaround! reform efforts, 
early results from an assessment of two cohorts of 
schools receiving SIG awards indicate only modest 
gains in student achievement for SIG-awarded schools 
overall. 


The lack of consistent, dramatic improvements in the 
lowest-performing schools reflects the severity of the 
school turnaround challenge, but also suggests the 
need to continuously evaluate and refine turnaround 
strategies. Studies indicate that it can take three to 
five years for turnaround efforts to impact overall 
student achievement, but leading indicators provide 
early evidence that can be used to gauge whether a 
school is on track to improve and to guide mid-course 
corrections that can increase the success rate of the 
turnaround effort. 


This brief examines current State education agency 
(SEA) practices for collecting and using leading 
indicators, describes how current practices can be 
strengthened, and identifies emerging promising 
practices in select States. This brief can be a useful 
resource for States working to improve their use 


For this brief, the term “turnaround" refers broadly to low- 
performing schools that are seeking dramatic improvements 
in student academic performance by implementing significant 
school improvement efforts, including schools that implement 
one of the four “turnaround models” defined by the U.S. 
Department of Education. 


°US. Department of Education. 2014. School Improvement Grant 
SIG) National Assessment Results Summary: Cohorts 1 and 2. U.S. 
Department of Education. Retrieved from: http://www?2.ed.gov/ 
programs/sif/assessment-results-cohort-1-2-sig-schools.pdf. 


> Kowal. J., & Ableidinger, J. (Public Impact). 2011. Leading Indicators 
of School Turnarounds: How to Know When Dramatic Change Is 
on Track. Charlottesville: University of Virginia's Darden/Curry 
Partnership for Leaders in Education. Retrieved from: http://www. 
DardenCurry.org. 
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of leading indicators to monitor and support 
improvements in low-performing schools. Key 
takeaways from the analysis of State practice include 
the following: 


- States collect a similar set of leading indicators 
to track progress, but they use this data in very 
different ways to monitor and support local 
educational agencies (LEAs) and schools. 


Many States try to track the quality of 
implementation for the purposes of monitoring 
and supporting LEA and school interventions,’ but 
few are collecting data and establishing processes 
to evaluate the impact of interventions on leading 
and long-term indicators for turnaround success. 


Leading indicator data are useful only when they 
are made actionable with strong routines for data 
analysis, planning and accountability. SEAs have 
established varying levels of rigor and consistency 
in their routines for monitoring turnaround 
progress and supporting LEAs and schools ina 
process of continuous improvement. 


SEAs can build the capacity of LEAs to support 
turnaround schools by establishing and modeling 
strong performance management practices in the 
collection and use of leading indicators. 


What are Leading Indicators? 


Leading indicators are performance measures that 
provide early signs for determining if an organization is 
on track for achieving a goal. In the school turnaround 
context, leading indicators, such as student attendance 
and observations of teachers implementing an 
instructional intervention, help school officials monitor 
implementation fidelity and early signs of progress in 
order to adjust strategy as needed. When these data 
are used to identify and address early implementation 


‘Note that the terms “interventions” and “strategies” are used 
somewhat interchangeably throughout the brief, with 
‘interventions’ often referring to school-level activities to improve 
the quality of school operations. 


> Kowal. J., & Ableidinger, J. (Public Impact). 2011. Leading Indicators 
of School Turnarounds: How to Know When Dramatic Change Is 
on Track. Charlottesville: University of Virginia's Darden/Curry 
Partnership for Leaders in Education. Retrieved from: 
http://www.DardenCurry.org. 


What is Performance Management? 


In an effort to support the success and sustainability of State turnaround efforts funded through the Race to 
the Top program, the Reform Support Network (RSN) began working with several SEA turnaround divisions 
in July 2013 to examine and strengthen their performance management practices. RSN defines performance 
management as a “systemic approach to ensure quality and progress toward organizational goals by methodically 
and routinely monitoring the connection between the strategies underway and the outcomes sought." The 
collaboration across States was guided by the RSN School Turnaround Performance Management 
Framework. The Framework is organized into four core components that establish a cycle of continuous 
improvement: goal setting, resource alignment, performance data tracking and accountability for results. 


Setop |: Charity rtop 1: 
ee ye eet and Cle 


While all variables in the Framework are critical to establishing an organization's performance management 
system, the collection and use of data to monitor progress against goals and evaluate the effectiveness 

of turnaround strategies allows SEAs, LEAs and schools to continuously refine and improve turnaround 
efforts, even in the face of shifting priorities, resources and performance expectations. 


problems and make mid-course corrections or intervene = + Acton early indicators of success or failure, 

when schools are not making adequate progress, implementing interventions as necessary to reach 
leading indicators have the potential to significantly your goals. 

increase the success rate of turnaround efforts. Existing 
research highlights four key strategies for using leading 
indicators to support low-performing schools: 


Collect a wide range of data and then narrow the 
most predictive data over time. ° 


. : © Kowal. J., & Ableidinger, J. (Public Impact). 2011. Leading Indicators 
Begin with known success factors and develop of School Turnarounds: How to Know When Dramatic Change Is 


leading indicators that have a demonstrated impact on Track. Charlottesville: University of Virginia's Darden/Curry 


on success. Partnership for Leaders in Education. Retrieved from: 


; ; http://www.DardenCurry.org. 
Monitor frequently to allow for mid-course . oe 


corrections. 
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Leading indicators are most effective when the SEA, 
LEA or school has established measureable goals for 
school turnaround that are aligned to a clear theory of 
action and corresponding set of focused interventions 
that address root causes of school performance 
challenges. For States, leading indicators can be an 
integral part of a strong performance management 
system that enables the SEA to both monitor progress 
against established school improvement goals 

and assess the extent to which specific turnaround 
strategies are accelerating progress toward these goals. 


Leading Indicators: 
What Nine States are Doing 


RSN identified nine States for analysis: Colorado, 
Delaware, Florida, Hawaii, New Jersey, New York, North 
Carolina, Rhode Island and Virginia. RSN facilitated 
conversations with SEA leaders and reviewed relevant 
documents in fall 2013 to assess each State's collection 
and use of leading indicator data. The conversations 
focused on three guiding questions: 


What leading indicators are SEAs using to track 
pre-conditions for student learning and student 
achievement against multi-year school turnaround 
goals? 

What indicators are SEAs using to track 


implementation of school turnaround strategies at the 
State, district and school level? 


What routines/processes are SEAs using to collect data, 
monitor performance and support interventions in 
turnaround schools? 


States Track Common 
Performance Indicators 


LEAs that administer SIG grants to low-performing 
schools are required to gather and report a number 
of leading indicators based on federal guidelines. 
Although there is variation across States, these 
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guidelines have led to a fairly consistent set of leading 
indicators that address the following four categories of 
school improvement: 


mproved student academic achievement, 
including interim assessments, standardized State 
test scores and college readiness indicators. 


mproved instructional quality, including teacher 
quality indicators, the percentage of students 
taught by highly effective teachers, the number of 
instructional minutes and the number of students 
enrolled in advanced or dual-enroll ment courses. 


Increased participation in school, including 
student/teacher attendance and teacher 
attendance, dropout rates and participation on 
State assessments. 


+ Improved school climate, including measures of 
discipline incidents, truancy, and levels of student, 
parent and teacher satisfaction. 


States are formally tracking and communicating 
statewide a very consistent set of leading indicators, 
largely influenced by federal SIG reporting 
requirements. A few of these leading indicators 

— such as student attendance, student discipline 
incidents and benchmark assessments — may show 
dynamic change during the school year that can 
reflect mid-year progress. However, most of the other 
indicators are fairly static during the school year and 
do not provide actionable data to inform mid-year 
adjustments. 


The table on the following page highlights the 
leading indicators identified across States, followed 
by brief descriptions of the four categories of leading 
indicators, with highlights of differentiated practice. 


Table 1: Leading Indicators Across States’ 


Leading Indicators co Joe fr | [we [we |wy [ri |v 


Student Academic Achievement 


State assessments** Vv All States ¥ 
Graduation rates** Vv All States 
nterim/benchmark assessments v v v v v v v 
9th-grade promotion 

ational norm-referenced tests (MAP, ACT, etcetera) v v v 
nstructional quality 
Distribution of teacher quality (based on State evaluation systems)* Vv All States ¥ 

umber of instructional minutes* Vv All States ¥ 
Participation in advanced or dual-enrollment courses* Vv All States ¥ 
Participation in school 
Student attendance* Vv All States ¥ 
Teacher attendance* Vv All States ¥ 
Dropout rates * Vv All States ¥ 
Participation in State assessments” Vv All States ¥ 
Chronic absenteeism v 
School Climate 
Student discipline incidents/suspensions* Vv All States ¥ 
Student truancy* Vv All States ¥ 
Parent, student and teacher satisfaction v va v we 
Teacher retention v 


* Required leading indicator for SIG reporting 
** Required lagging indicator for SIG reporting® 


measure improvements in academic achievement 
during the school year. Many conduct some form 

of quarterly monitoring reviews, and results from 
benchmark assessments are frequently a primary focal 
area for assessing progress. While most States do not 
mandate the use of specific interim benchmarks in 
low-performing schools, many schools incorporate 
these assessments into their State-monitored 

school improvement plans. New Jersey has taken 

a somewhat unique approach by requiring all 


Student academic achievement 


While student achievement indicators such as State 
test scores and college readiness indicators (for 
example, graduation rates, college matriculation rates) 
are most often associated with lagging indicators of 
student outcomes, they also can represent leading 
indicators of turnaround progress, demonstrating 
“quick wins’ that are associated with success. Some 
States also have incorporated 9th-grade promotion 
rates and performance on the ACT assessment for high 
schools as important, interim metrics to predict high 
school graduation and college enrollment outcomes. 


Inventory of leading indicators collected by States based on 
interviews with SEA administrators and review of SEA websites. 


8 Guidance on fiscal year 2010 School Improvement Grants under 
section 1003(g) of the Elementary and Secondary Education Act 
of 1965. U.S. Department of Education, March 1, 2012. Available at: 
http://www?2.ed.gov/programs/sif/sigguidance03012012.doc. 


Most States also review interim benchmark assessment 
data as a leading indicator for school progress to 
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Priority Schools? to implement interim benchmark 
assessments, and by providing a benchmark testing 
system for these schools. Priority Schools and LEAs can 
choose to opt out and use their own assessments, but 
most choose to implement the State-supported tests. 
While implementing uniform benchmark assessments 
across Priority Schools can provide better cross-State 
data for SEA monitoring and improvement planning, it 
also increases the burden on the SEA. The State must 
provide high-quality assessments that are aligned to 
the State standards and school curriculum, and an 
adequate data reporting system to facilitate school- 
level data analysis and action planning. 


Instructional quality 


As required by SIG, all States track data related to 
increased learning time (instructional minutes), 
distribution of teachers by “effectiveness ratings” and 
advanced course completion numbers. 


Increased learning time is a requirement for 

the implementation of the “transformation” and 
“turnaround” models defined by the U.S. Department 
of Education.'° Federal SIG guidance" requires 
additional time for 1) instruction in core academic 
subjects, 2) additional time for other subjects and 
provision of enrichment activities for students, and 

3) additional time for teacher collaboration, planning 


"Priority School” is defined by the U.S. Department of Education 
though the ESEA flexibility wavier as “a school that, based on 
the most recent data available, has been identified as among 
the lowest-performing schools in the State. The total number of 
Priority Schools in a State must be at least five percent of the Title 
| schools in the State.” Based on federal guidelines, States establish 
criteria for identifying and supporting Priority Schools. See ESEA 
Flexibility Policy Document at: http://www2.ed.gov/policy/elsec/ 
guid/esea-flexibility/index.html. 


Schools designated as low-performing through the Race to the 
Top and SIG programs were required to implement one of four 
school intervention models: Turnaround, Restart, School Closure 
or Transformation. Note, additional models were added in 2015 to 
include a State-determined model, evidence-based, whole school 
reform model and early learning model (See reference below). 


"U.S. Department of Education. 2015, March. Guidance on School 
Improvement Grants (p. 1). Washington, DC: Office of Elementary 
and Secondary Education. Retrieved from: http://www2.ed.gov/ 
programs/sif/sigguidance032015.doc. 
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and professional development. Although all States in 
this report track the provision of increased learning 
time as an indicator for low-performing schools, they 
do not systematically track more detailed information 
about the specific use of increased learning time, nor 
its impact on the quality of teaching and learning. In 
this sense, the metric primarily serves a compliance 
function for adherence to required turnaround models, 
rather than as a leading indicator to assess turnaround 
progress. North Carolina provides a notable exception 
and requires LEAs and schools to identify specific goals 
for increased learning time for each of the three focal 
areas (for example, core subjects, enrichments, teacher 
collaboration). 


Student enrollment and completion of Advanced 
Placement (AP), International Baccalaureate (IB) and 
dual-enroll ment coursework are widely recognized as 
relevant measures to determine if high school students 
are exposed to rigorous academic content that will 
prepare them for success in college. The intensity and 
quality of a student's high school curriculum is a strong 
predictor of bachelor’s degree completion." All States 
report data on this metric as a reporting requirement 
of SIG implementation. 


Teacher effectiveness measures, which have 

just recently become available as SEAs and LEAs 
implement new systems for teacher evaluation and 
support, are a potentially valuable leading indicator 
for school turnaround progress. Teacher effectiveness 
is measured differently in each State and is usually 

a composite rating of multiple measures, including 
student academic growth, individualized performance 
goals and qualitative measures of teacher practice 

(for example, observations and surveys). The Race 

to the Top program and the ESEA flexibility waivers 
created an impetus for the majority of States to enact 
significant reforms to teacher evaluation systems since 
2010. Research indicates that teacher quality is the 


2 Adelman, C. (1999). Answers in the tool box: Academic intensity, 
attendance patterns, and bachelor’s degree attainment. Washington, 
DC: U.S. Department of Education, Office of Educational Research 
and Improvement. (pp. 8-9) Retrieved from: 
http://eric.ed.gov/?q=Academic+Intensity%2c+Attendance+Pa 
tterns&id=ED43 1363. 


most significant factor to affect student learning." 
Teacher effectiveness ratings, designed to measure the 
impact of teaching on student learning, can therefore 
be an important indicator for assessing whether a 
school turnaround effort is on track. But as a leading 
indicator, its utility to guide mid-course corrections 

is compromised because the metric is typically 
measured and reported annually and thus remains 
static over the course of the school year. Furthermore, 
depending on the frequency at which teachers are 
evaluated, school measures of teacher effectiveness 
can remain relatively unchanged from year-to-year 
unless associated with dramatic turnover of staff. 


In addition to measuring teacher effectiveness, States’ 
guidance on the school improvement planning 
process typically requires schools to evaluate and 
identify improvements for strengthening the quality 
of school leadership. However, none of the States 
profiled in this brief collect specific indicators related 
to principal effectiveness, nor is this a SIG reporting 
requirement. This is a notable absence given that a 
highly effective principal can contribute an additional 
seven months of learning in a single academic year 
versus an average principal, that school leadership 
factors in to most federal turnaround models and that 
many SEA turnaround units cite “effective leadership” 
as a primary component of their theory of action for 
school turnaround. Although States have been quicker 
to adopt and implement new teacher evaluation and 
support systems, many States also are implementing 
new principal evaluation and support systems that can 
provide more actionable data for assessing principal 
effectiveness in turnaround schools. However, these 
new systems will take time to become valid and 
reliable measures of principal quality, and may not 

be sufficient to assess the leadership competencies 
and actions necessary for success in a turnaround 
environment. 


One report on leading indicators in school turnaround 
identifies a range of indicators that could be 

used to assess research-based turnaround leader 
competencies and actions that are correlated with 


"3 Eric A. Hanushek, Steven G. Rivkin, and John F. Kain, “Teachers, 
Schools, and Academic Achievement,’ Econometrica, 73, 2 
(2005), 417-458. Available at: http://www.econ.ucsb.edu/~jon/ 
Econ230C/HanushekRivkin.pdf. 


turnaround success.'* SEAs and LEAs may consider the 
extent to which their principal evaluation system can 
provide accurate and early indicators of turnaround 
progress, and if supplementary measures, such as 
demonstrated evidence of “turnaround leader actions,’ 
can better assess principals’ ability to lead dramatic 
school improvement. 


School climate 


Successful turnaround leaders often identify the 
transformation of school climate and culture as a 
prerequisite for achieving dramatic and sustainable 
improvements in student academic achievement. 
States appear to collect a similar set of leading 
indicator data related to school climate, and 
consistently track metrics related to student/teacher 
attendance, student drop-out rates, and student 
discipline incidents and truancy. Our brief scan of 
SEA-collected data indicate that many of these metrics 
change significantly over the course of the turnaround 
effort, and therefore provide dynamic and actionable 
data for tracking school and LEA turnaround progress. 


Several States require turnaround schools to collect 
survey data from staff, parents and/or students to 
assess stakeholder satisfaction and conditions for 
teaching and learning in schools. Both Colorado and 
Delaware administer the Teaching, Empowering, 
Leading and Learning (TELL)" survey of teachers 

to assess teaching conditions at the school, district 
and State level. Although neither State identifies 
performance targets for survey data, survey results are 
reviewed as part of the SEA performance monitoring 


“Kowal. J., & Ableidinger, J. (Public Impact). 2011. Leading Indicators 
of School Turnarounds: How to Know When Dramatic Change Is 
on Track. Charlottesville: University of Virginia's Darden/Curry 
Partnership for Leaders in Education. Retrieved from: 
http://www.DardenCurry.org. 


The TELL survey is developed and supported by the New Teacher 
Center (http://www.newteachercenter.org). The New Teacher 
Center works with more than a dozen States to customize and 
administer anonymous, online surveys to all teachers, principals 
and other licensed educators in the State. The TELL survey is an 
anonymous statewide survey of licensed school-based educators 
to assess teaching conditions at the school, district and State 
level. With the leadership of Governor Jack Markell, Secretary Mark 
Murphy, and the Delaware Department of Education, a coalition 
of education stakeholders, working with the New Teacher Center, 
administered the Delaware Teaching, Empowering, Leading and 
Learning (TELL Delaware Survey) from January 22 to February 25, 
2013. 
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routines with low-performing schools. Florida 

and lllinois also administer a statewide educator 
survey, the 5 Essentials Survey,'® to all schools and 
incorporate survey data into SEA monitoring routines 
for low-performing schools. New Jersey administers 
parent, teacher and student satisfaction surveys in 
Priority Schools that include performance targets 

for survey participation (95 percent for teachers and 
students and 25 percent for parents) and stakeholder 
satisfaction (80 percent of responders indicate their 
school “meets expectations’). 


Teacher retention and turnover are other important 
indicators of school climate and culture. However, 
surprisingly few States collect data on teacher 
retention or on retention by teacher effectiveness 
ratings. Although school improvement efforts often 
prioritize recruitment of and professional development 
for teachers, recent studies suggest that highly 
effective and ineffective teachers typically leave 
schools at about the same rate.'’ A brief from the RSN 
highlights emerging State practices for incorporating 
“selective retention” as a measure of principal 
effectiveness.'® Such measures can also be a useful 
leading indicator for assessing the extent to which 
schools are establishing a culture that retains talented, 
highly effective faculty. 


States Use Many Approaches 
to Monitor Implementation 


SEAs focus significant time and resources to ensure 
that LEAs and schools are implementing turnaround 
strategies that comply with grant requirements and 
mandated State requirements for low-performing 
schools. Compliance monitoring is a traditional role 
of the SEA, and it continues to be an important and 
necessary requirement for ensuring transparency 

n the use of public dollars. This form of monitoring 


'eThe Illinois 5 Essentials Survey (http://www.isbe.net/Sessentials/ 
default.htm) is developed and administered to all Illinois schools 
on behalf of the Illinois State Board of Education. Other States, 
including Florida, have adopted the tool. 


"The New Teacher Project (TNTP). 2012. The Irreplaceables: 
Understanding the Real Retention Crisis in America’s Urban 
Schools. Brooklyn, N.Y.: TNTP. Retrieved from: http:// 
tntp.org/irreplaceables. 


‘Reform Support Network (2015). Incorporating Retention of 
Effective and Highly Effective Teachers in Principal Evaluations. 


(Washington, D.C). http://www2.ed.gov/about/inits/ed/implementation- 


support-unit/tech-assist/incorporatingretention.pdf. 
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allows States to ask LEAs, “Did you do what you 

said you were going to do?” Beyond compliance, 
monitoring implementation can also help States 
evaluate the extent to which strategies might correlate 
to reaching student achievement goals. Measuring 
and monitoring implementation can help States and 
schools determine whether the strategies in place 

are in fact helping the school make progress toward 
intended goals. 


In order for these implementation measures to be 
useful leading indicators, they must not only measure 
compliance with required and planned activities, they 
must also answer the following questions related to 
the quality and impact of strategies: 


1. Were the strategies implemented with fidelity? 
(inputs/fidelity measures) 


2. Did the implemented strategies lead to positive 
changes in adult behaviors or practices? (outputs/ 
quality measures) 


3. Did the strategies lead to measurable 
improvements in student behavior and/or 
achievement? (outcomes/impact measures) 


Monitoring fidelity of implementation 


All States track district and school fidelity in 
implementing school improvement strategies. The 
intensity and routines for monitoring vary across 
States. But each State has established a process 

to determine if schools are implementing the 
interventions identified in their school improvement 
plans (or SIG/Race to the Top applications), and 

to what extent federal and State funds are used 

for activities identified in the school improvement 
plan and/or grant application. The Delaware School 
Turnaround Unit adopted a monitoring system for 
Priority Schools in 2013-2014 that uses a Red/Yellow/ 
Green implementation status update that the SEA 
collected quarterly or bimonthly. In previous years, 
these data were only collected at the end of year, 
limiting the SEA’s ability to collaborate with schools 

in mid-year corrections. North Carolina and Virginia 
are two of 24 States plus the District of Columbia that 
use Indistar®,'? a Web-based tool, to support planning, 


'8What Is Indistar? Retrieved from: http://www. indistar.org/whatis. 


implementation, monitoring and coaching on school 
improvement activities. The tool requires schools to 
prioritize and establish detailed implementation plans 
tied to “indicators of evidenced-based practices at 
the district, school and classroom level to improve 
student learning.’ Although the assessment and 
tracking of implementation progress is primarily 
conducted at the school level, as a Web-based tool, 
Indistar® helps the SEA track progress, provide remote 
feedback and facilitate meetings focused on effective 
implementation of turnaround strategies. 


Measuring quality of implementation 
and impact on performance outcomes 


While States have adopted a wide range of practices 
to measure compliance in implementing school 
turnaround strategies, most States have not 


established explicit metrics or systematic processes 
that attempt to measure the impact of strategies on 
school turnaround goals. If“outcomes’ are the ultimate 
student achievement goals established for turnaround 
schools, “outputs” can be used to define the changes in 
instructional practice, school climate or other leading 
indicators that are achieved through implementation 
of turnaround strategies, and which may ultimately 


lead to 
are the 


improved student learning (outcome). “Inputs” 
intervention strategies and resources that 


are implemented to improve the leading indicators 
associated with ultimate student achievement 
outcomes. For example, new selection practices 
(input) for hiring teachers in a turnaround school are 
intended to increase the number of highly effective 
teachers in the school (output), which should result in 
higher student achievement (outcome). 


a ee ot 


Designed to track ... Fidelity of Implementation 


Answer question of ... Was the strategy 


implemented? 


Implementation of new 
teacher selection practices for 
turnaround schools. 


Example 1: Teacher selection 


Number of minutes of 
professional development 
provided to teachers and staff 
during the school year. 


Example 2: Behavior 
management professional 
development 


Quality 


Did the strategy result in 
changes to adult behaviors? 


Increase the number of highly 
effective math teachers in the 
school. 


Staff survey on the 
effectiveness of professional 
development 

and 

Observed changes in teacher 
response to student behavior. 


Impact 


Did the strategy lead to 
measurable improvement 
in student behavior or 
achievement? 


Increase in student 
achievement in mathematics 
on State assessments. 


Decrease in number of student 
discipline referrals. 
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Identifying clear outputs and outcomes for turnaround 
strategies is not always so straightforward. Several 
State turnaround leaders identified this as a next step 
in strengthening performance management practices. 
Leaders in Delaware said that linking strategies to 
improvement metrics was more manageable with 
Focus Schools,”? where interventions are targeted 

to reducing achievement gaps for specific student 
subgroups. Making causal connections in their Priority 
Schools is more challenging because schools are 
implementing a much wider range of school-wide and 
targeted school improvement strategies and looking 
for measurable improvements across a broader range 
of indicators and student subgroups. 


Virginia also cited “output” measures linked to 
turnaround strategies as an area for development. 
While Indistar® helps the State monitor detailed 
information about fidelity of strategy implementation, 
the implementation data are not explicitly linked to 
measureable improvements in instructional practice 
or other leading indicators that signal progress toward 
goals. 


Further complicating the challenge of linking 
intervention “inputs” to “outputs” is the sheer volume of 
interventions that SEAs implement and track. SEAs and 
LEAs are addressing this challenge by encouraging 
schools to narrow in on a discrete set of interventions 
and corresponding leading indicators, and by 
explicitly linking interventions to output indicators 
and measures of student outcomes. In Florida, for 
example, the Bureau of School Improvement modified 
its SIG application and monitoring process to narrow 
the number of turnaround interventions that are the 
primary focus for LEAs and schools. Florida identified 
five “Areas of Focus” from the U.S. Department of 
Education requirements that the State identified from 


?0"Focus School’ is defined by the U.S. Department of Education 
though the ESEA flexibility wavier as “a Title | school in the State 
that, based on the most recent data available, is contributing 
to the achievement gap in the State. The total number of focus 
schools in a State must equal at least 10 percent of the Title | 
schools in the State’ Focus Schools have the “largest within-school 
gaps between the highest-achieving subgroup or subgroups and 
the lowest-achieving subgroup or subgroups.’ Based on federal 
guidelines, States establish criteria for identifying and supporting 
Focus Schools. See ESEA Flexibility Policy Document at: http:// 
www2.ed.gov/policy/elsec/guid/esea-flexibility/index.html. 


research and experience as being vital to continuous 
school improvement.?! Focal areas include multi-tiered 
system of supports, standards-based instruction, 
increased learning time, data-differentiated instruction 
and professional development. The SIG application 
process directs LEAs and schools to provide specific 
action plans and performance targets about the 
implementation of interventions aligned to these 

focal areas, and deprioritizes planning and reporting 
associated with other interventions required in the 
federal school turnaround models. 


Rhode Island’s quarterly monitoring process of School 
Reform Plans (SRPs) represents a very intentional 
approach to assess the impact of school turnaround 
strategies. (See excerpt of Rhode Island SRP Process 
for Tracking Impact of Turnaround Strategies). For 
each major improvement goal established in the SRP 
(for example, increase student proficiency rates on 
reading assessment), schools must identify a discrete 
set of interventions/“inputs” and a corresponding set 
of implementation process leading indicators (adult 
behavior “outputs”), and impact leading indicators 
(student ‘outcomes’) that are expected to occur 

as a result of the intervention. Schools establish 
specific metrics for these indicators and self-assess 
progress. Each quarter, schools submit data from the 
measurement tools to help the school, LEA and SEA 
understand whether the behaviors and outcomes are 
occurring with the frequency and quality intended. 


Colorado also differentiates leading indicators based 
on inputs, outputs and outcomes for turnaround 
intervention strategies. As part of the school 
improvement planning process, Colorado schools are 
required to identify “implementation benchmarks” for 
each improvement strategy that includes indicators 
to measure fidelity of implementation (for example, 
professional development sessions held) and impact 


*1 Florida Department of Education, “School Improvement 
Grant Proposal Design and Structure Presentation.’ Accessible 
through the School Turnaround Performance Management 
Toolkit at: https://rtt.grads360.org/?p=rtt#communities/pdc/ 
documents/7556. 


22 See Rhode Island's Facilitator's Guide to Quarterly Monitoring 
of School Reform Plans For Priority and Focus Schools (October 
2014): http://www.ride.ri.gov/Portals/0/Uploads/Documents/ 
Information-and-Accountability-User-Friendly-Data/ 
Transformation/Facilitators-Guide-Quarterly-Monitoring- 
Fall-2014.pdf. 
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Improvement Goal: 


By 2015, 60 percent of all students will meet or exceed a student growth percentile (SGP) ranking of 50 according to 
the STAR Reading Assessment and of the remaining students, 75 percent will have an SGP between 35 and 49. 


Strategy / Intervention (“Input"): 


Our reading coaches and English Language Arts (ELA) professional development will improve STAR reading 


proficiencies through three major strategies: 


Lesson Planning: Coaches will review lesson plans for observed lessons, scoring them on a school-designed 
Lesson Plan Rubric from 1 to 5, with 5 being the highest quality. Coaches will work with teachers as needed, one- 


on-one to improve lesson plan quality. 


Behavior Management: Conduct school-wide professional development on improving student behavior by 
using effective behavior management strategies. Coaches will record incidents of misbehavior and how often 
those misbehaviors are effectively responded to. 


Research-based instructional strategies: Conduct ELA-specific professional development sessions during 
common planning time to focus on discrete set of Marzano's research-based instructional strategies, to be 


determined on a quarterly basis. 


Implementation Process Leading Indicator 
(Adult Behavior “Output”)—for Lesson Planning 
intervention 


Increase average scores on lesson plan rubric. 


on adult behaviors (for example, new instructional 
strategy implemented). To measure student outcomes, 
schools are required to identify “interim measures” 

for each major performance goal that are based on 
local data and are available at least twice during the 
year to measure process against annual performance 
targets. Teams review implementation benchmarks 
and interim measures at least quarterly to determine if 
“improvement strategies are being implemented with 
fidelity and are having the desired effects’ 


States Use the Information 
in Different Ways 


While States differ somewhat in the range of leading 
indicators collected for tracking turnaround progress 


3 See Colorado Department of Education's Unified Improvement 
Planning Handbook: http://www.cde.state.co.us/uip/uip_ 
handbook_v40. 


Impact Leading Indicator (Student Outcome)—for 
Lesson Planning intervention 


For teachers with lesson plans scoring 4 or higher on 
lesson plan rubric, percentage of their students on target 
to meet STAR growth goals. 


and strategy implementation, SEAs vary significantly in 
how they use these indicators to monitor, support and 
intervene in school turnaround efforts. 

The RSN School Turnaround Performance 
Management Framework includes a range of 
evidentiary statements on effective practices in the 
collection and use of performance data, and highlights 
the need for SEAs to establish regular routines and 
systems that: 


- Align the data collection process with routines for 
monitoring progress; 


+ Facilitate discussions with LEAs and schools on the 
impact of turnaround strategies against leading 
indicators and long-term student outcome goals; 


+ Support LEAs in the improvement of their own 
performance management routines. 
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The States analyzed in this report use various 

methods to collect and analyze leading indicators. 
Some States utilize frequent, monthly meetings to 
review data progress with LEAs, while others use 
bi-annual meetings. The depth of analysis also 

varies, with some States primarily focused on fidelity 
measures and others utilizing structured protocols 

for reviewing academic benchmark data and other 
indicators of intervention progress. This variation is 
driven by different levels of SEA capacity, differences in 
underlying strategies for supporting low-performing 
schools, and the extent to which the SEA has 
incorporated strong performance management 
practices into its system of monitoring, support and 
accountability. Several of the States profiled in this 
brief provide useful examples about challenges and 
promising practices for establishing strong monitoring 
routines. 


Aligning data collection processes with 
SEA routines to monitor progress 


For several States in this brief, the quality and 
consistency of progress monitoring routines are 
compromised by the SEAs' limited access to timely, 
quality data. For example, as part of Colorado's Unified 
Improvement Plan process, for example, schools and 
LEAs are required to set targets for implementation 

of major improvement strategies. However, the 
systems for tracking implementation are focused 

on individual school tracking, and the SEA does 

not have a centralized database or data collection 
process that aggregates data across LEAs and schools. 
The SEA reports that some LEAs have established 
performance management practices that incorporate 
structured routines to regularly (monthly) collect and 
analyze implementation data and interim student 
measures. However, the SEA has played a limited role 
in conducting or influencing LEA adoption of strong 
monitoring routines, in part because the relevant 
data is simply not available to engage in these 
conversations. Starting in the 2014-2015 school year, 
Colorado addressed this data gap challenge for a 
subset of Priority Schools. Colorado schools that opt-in 
to Colorado's Network Schools initiative will submit 
self-reported data on turnaround progress through 
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an online platform to guide monitoring and support 
conversations with the SEA. 


In New Jersey, the SEA established Regional 
Achievement Centers (RACs) in 2012-2013 to provide 
school improvement monitoring and support to the 
State's Priority and Focus Schools. During the launch 
of the RAC system of support, the State also was 
implementing a data collection and analytics software 
to support the creation and tracking of school 
intervention plans. The system was designed to link 
interim assessment data with school improvement 
activities in order to enable schools and RACs to 
assess the impact of turnaround interventions on 
eading indicators of student outcomes. However, 
delays and problems with system development and 
implementation significantly restricted access to timely 
and accurate data, and compromised the ability of the 
RACs to establish effective monitoring routines. 


New York's engagement with the RSN Performance 
Management workgroup focused on establishing 
strong performance management routines to 
intentionally shift SEA-LEA conversations from a 
compliance exercise to data-focused problem- 
solving meetings that rigorously tracked progress 
toward goals and assessed the impact of turnaround 
strategies. The shift in focus has required a significant 
change in mindset for both the SEA and LEAs, but 
also exposed gaps in the data that the SEA collects 
and reviews from LEAs and schools. In order for the 
SEA to implement data review and analysis protocols 
with the LEAs, it first created internal shared online 
spreadsheets that SEA staff used to populate LEA 
and school-level, longitudinal data from disparate 
sources (including State system and LEA-provided data 
points.) While this approach provided a short-term 
solution to support the new data review routines, New 
York is now building a Web-based platform that will 
facilitate direct LEA input of data to provide a more 
sustainable model for the State. 


For Virginia and North Carolina, the Indistar® system 
provides a ready-made solution for tracking a large 
amount of data to monitor implementation of 


See New York State Education Department's Web-Based Data 
Collection Tool: https://docs.google.com/spreadsheets/d/1 
VZ2w6QGuB4dnLSYZ82isMwX70B0Vnwg7WxyKbhtxqVA/ 
edit#gid=0. 
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turnaround interventions. Coupled with effective State 
data warehouses, the challenge to implementing 
strong performance management routines may not 
be the absence of data, but perhaps the ability to 
focus on the highest priority indicators and establish 
routines with schools and LEAs that facilitate problem- 
solving to meet goals. North Carolina recognized this 
challenge in describing its lessons learned about the 
importance of establishing clear targets for school 
improvement, and narrowed its focus with LEAs and 
schools to the highest-priority interventions and 
indicators. North Carolina first identified a subset of 28 
indicators (from the 80+ indicators within the Indistar® 
system) that schools need to implement over a 3-year 
school improvement cycle, and further narrows its 
focus by concentrating SEA feedback on 3-5 indicators 
that are prioritized by schools for the academic year. 


Facilitating discussions about the impact 
of turnaround strategies on leading 
indicators and long-term goals 


While few States have explicitly and consistently 
incorporated implementation “outputs” into their data 
collection processes, several have established data 
monitoring routines that are designed to replace the 
SEA's traditional role of compliance monitoring with 

a new focus on helping LEAs and schools answer 

the question, “Did our turnaround strategies have 

a measurable impact that will result in improved 
outcomes for students?” 


Rhode Island, for example, has established promising 
practices for facilitating these discussions with LEAs 
and schools as part of its “Quarterly School Monitoring” 
meetings. The facilitator’s guide to the quarterly 
meetings” is driven by the specific objectives to 
“increase understanding of the relationship between 
current measures of performance indicators and 

the processes of implementation” and to “identify 
and prioritize next steps to better assess and/ 

or better implement interventions.” Rhode Island 
implemented new protocols and data collection 
processes during the 2013-2014 school year, with 


5 See Rhode Island's Facilitator's Guide to Quarterly Monitoring 
of School Reform Plans For Priority and Focus Schools (October 
2014): http://www.ride.ri.gov/Portals/0/Uploads/Documents/ 
Information-and-Accountability-User-Friendly-Data/ 
Transformation/Facilitators-Guide-Quarterly-Monitoring- 
Fall-2014.pdf. 


the intent of increasing the level of collaboration, 
mutual accountability, and LEA capacity to analyze 
and respond to progress monitoring data. The SEA 
took the additional step of internally evaluating how 
effectively the new routines were meeting established 
objectives, and worked with the RSN to conduct 

an external observation and review of the routines. 
The external review used a scoring rubric to assess 
different dimensions of the meeting protocol, with 
the goal of supporting continuous improvement of 
SEA-LEA school data routines. The external review 
identified many strengths, as well as some areas for 
improvement that could be broadly applied to States 
looking to implement stronger data routines: 


- Focus on two to three strategies per school for 
the entire year. It can be difficult to understand 
progress over time at schools if the focus of the 
routine changes quarter to quarter. Aim to focus on 
a consistent set of strategies (up to three) for each 
routine at the school level. This will help to focus 
the conversation around implementation, evidence 
and problem-solving for those specific strategies. 


« Focus on effective data use in schools. Strive for 
effective data use to inform implementation of 
the two to three strategies at the school level, not 
just to inform the routine. If the school is using 
data effectively at the teacher and leadership 
team levels, the school and district will then have 
more accurate and robust data available to inform 
ongoing performance management routines. 


* Train and follow a standard set of effective 
facilitation practices. Strong facilitation practices 
can enhance the school experience at the meeting 
(for example, practices such as posting objectives, 
reviewing next steps and encouraging co- 
facilitation of meetings). 


+ Model strong problem-solving. Incorporate 
intentional and structured problem-solving 
protocols into the monitoring routines of all schools 
and use end-of-year meetings to set focus areas for 
the start of the following year. 


Helping LEAs improve their own 
performance management routines 


Several States view the development of effective 
data-monitoring routines as serving multiple purposes: 
improving the SEAs’ processes for monitoring, 
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supporting LEA and school-level turnaround efforts, 
and strengthening the performance management 
capacity of LEAs. Most of the States analyzed for this 
brief have incorporated “LEA capacity building” as an 
explicit component of their theory of action for school 
turnaround. This has been a shift for some States, as 
their experience with implementation of Race to the 
Top and SIG initiatives have underscored the essential 
role that LEAs must play in supporting low-performing 
schools, and as SEAs plan for reduced resources with 
the phase-out of federal grants. 


Florida, for example, has taken a particularly robust 
approach to building the performance management 
practices of LEAs in support of school turnaround. 
Florida's strong emphasis on continuous improvement 
(performance management) is guided by a focused 
theory of action: “/fwe increase the automaticity and 
efficacy with which districts and schools problem-solve 
around barriers to improved student achievement, then 
outcomes for all Florida students will improve. Florida's 
Bureau of School Improvement is implementing 

its theory of action by training LEA leaders on an 
Eight-Step Planning and Problem Solving process” 
to guide the development and implementation of 
SIG applications, including a multi-day professional 
development convening of LEA leaders. 


Conclusion 


The lack of consistent, dramatic improvements in the 
lowest-performing schools reflects the severity of the 
school turnaround challenge, but also suggests the 


6 Florida Department of Education, “A Guide to Eight-Step Planning 


and Problem Solving.’ Accessible through the School Turnaround 
Performance Management Toolkit at: https://rtt.grads360. 
org/#communities/pdc/documents/7555. 


need to continuously evaluate and refine turnaround 
strategies. Leading indicators can provide early 
evidence to measure progress against ambitious 
school turnaround goals and identify opportunities for 
mid-course corrections that can accelerate the rate of 
turnaround success. 


While States annually track and report a similar set of 
leading indicators, there is significant differentiation in 
how SEAs track implementation and impact of school 
improvement interventions, and in the routines that 
SEAs regularly use to engage with LEAs and schools 
to evaluate school progress and facilitate continuous 
improvement. The RSN’s work with SEA turnaround 
divisions to strengthen performance management 
practices resulted in several States adopting more 
rigorous and intentional practices for using leading 
indicators to support problem-solving and data-driven 
decision-making. 


While there is much room for all States, districts and 
schools to improve their practices, several States 
analyzed in this brief have established promising 
performance management practices that provide 
useful insights and practical tools to guide other States 
in the years ahead. 


For additional resources to support States’ 
performance management practices, see the RSN's 


School T urnaround 
Performance 
Management Toolkit 
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